Transmitting high-quality, real-time interactive video over lossy networks, such as the Internet and wireless networks, is very challenging. Because of limited bandwidth on networks and the bandwidth-intensive nature of video, video transmission requires extremely high compression efficiency. However, state-of-the-art compression standards (MPEG, H.261) are not designed for transmission over a lossy channel. Although they can achieve very impressive compression efficiency, even small data losses can severely degrade video quality. A few bit errors in encoded data can cause the decoder to lose synchronization in the encoded stream and can render useless all the data received until the next synchronization point. Furthermore, motion estimation and compensation in these codecs pose an even more severe problem, namely, error propagation (or error spread). Motion estimation removes temporal redundancy in successive video frames (inter frames) by encoding only pixel value differences (prediction error) between a currently encoded image and a motion-predicted image created from a previously encoded image (reference frame). Image distortion in a reference frame can propagate to its succeeding frames and becomes amplified as more bits are lost.
Conventional work on loss recovery focuses on repairing packet losses before the scheduled display times of those video frames contained in lost packets. However, this approach is ineffective for interactive video because data losses inevitably occur in packet-switched communication, and detecting and repairing losses causes latency. To handle this latency, existing techniques introduce additional delays in frame display times. However, delaying frame playout times greatly impairs the interactiveness of video communication.
Many researchers have proposed using retransmission of lost packets by delaying frame playout times to allow arrival of retransmitted packets before the display times of the video frames associated with the packets. Any packet received after the display time of its associated video frame will be discarded. In these schemes, the display time of a frame is delayed by at least three one-way trip times after its initial transmission (two for frame transmission and one for a retransmission request). This latency can significantly impair the interactiveness of any video applications under the current Internet.
Forward error correction (FEC) is commonly proposed for error recovery of continuous media transmission. However, conventional FEC schemes do not work well for interactive video. The reason that conventional FEC schemes do not work well with interactive video is that unless the playout time of a frame is delayed, both the original packets and their parity or FEC packets must be transmitted within the same frame interval, rendering the schemes very susceptible to burst loss. Moreover, since FEC is applied to a block of packets, before FEC packets are computed and transmitted, a large delay can occur.
Commonly-assigned, co-pending U.S. patent application Ser. No. 09/079,621, filed May 15, 1998 now U.S. Pat. No. 6,104,757, and entitled xe2x80x9cSYSTEM AND METHOD OF ERROR CONTROL FOR INTERACTIVE LOW-BIT RATE VIDEO TRANSMISSIONxe2x80x9d (hereinafter, xe2x80x9cthe RESCU Patent Applicationxe2x80x9d) discloses a complementary approach to the above-mentioned approaches by focusing on eliminating error propagation when distortion on displayed images occurs. This approach is referred to as recovery from error spread using continuous updates (RESCU). The point of departure from existing approaches described in the RESCU Patent Application is that packets do not have to arrive in time for them to be xe2x80x9cusefulxe2x80x9d for display of that video frame. Of course, if packets can arrive before the display times of their frames, that is optimal. However, due to packet losses and high latency, repair packets inevitably arrive xe2x80x9clate,xe2x80x9d causing distortion in displayed images, which can propagate to successive frames. These late repair packets can be used to stop error propagation. In motion-compensated codes, the correct display of a frame depends on the successful reception of all of its reference frames. If displayed frames are buffered and late packets are used to restore errors in the buffered frames, error propagation can be stopped. The reason that error propagation can be stopped is that the buffered frames will be used as reference frames for later frames.
RESCU has been shown to be effective for interactive video transmission when retransmission is used to recover lost packets and round trip delays are small. Retransmission tends to prolong error propagation because of the delay involved in detecting and retransmitting lost packets. Moreover, in some networks, such as wireless, cable modems, and direct satellites, feedback channels are highly contentious, and bandwidth-limited. Thus, in these networks, frequent transmission of feedback to the sender is too expensive. Accordingly, there exists a need for improved methods and systems for performing error recovery when transmitting compressed video over a lossy packet based network.
According to one aspect, the present invention includes a new FEC technique for interactive video that combines FEC with RESCU. By incorporating this FEC technique, RESCU can perform very effectively in an environment where little or no feedback is available, or transmission delay is too high for retransmission to be effective. The FEC scheme according to the present invention clearly differs from the conventional schemes in that FEC packets can be transmitted over a longer period than a single frame interval without introducing delay in frame playout times. Since RESCU uses FEC packets to restore buffered reference frames (referred to herein as periodic frames), FEC packets can be transmitted over a relatively longer period, interleaving with the packets of other (non-periodic) frames to help reduce the effect of bursty losses. This interleaving is different from link-level symbol interleaving where symbols from multiple codewords are interleaved. The granularity of interleaving according to the present invention is much larger and thus, more effective than link-level interleaving. Since RESCU makes non-periodic frames temporally depend only on the immediately preceding periodic frame and, unlike retransmission, FEC involves no feedback delay, the proposed technique incurs shorter recovery delays and accordingly shorter error propagation. Thus, the present invention can be effective for high frame rate transmission over lossy, high-latency networks.
Accordingly, it is an object of the present invention to provide improved methods and systems for reducing error spread in video transmission over a packet-based network.
Some of the objects of the invention having been stated above herein, other objects will become evident as the description proceeds when taken in connection with the accompanying drawings as best described herein below.